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TITLE OF THE INVENTION 

IMAGE PRODUCING DEVICE AND IMAGE DEVIATION 
AMOUNT DETECTION DEVICE 

5 BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates to a technique of producing image 
data from video information comprising a numerous sets of frame 
information. 

10 2. Description of the Related Art 

[0002] Conventionally, image data which expresses still images has been 
produced from video information expressing video images acquired by 
digital video or the like, and still images have been printed on the basis of 
this image data. Video information is constructed from numerous sets of 

15 frame information that express images by means of halftone data for 
numerous individual pixels, and image data which expresses still images in 
halftones by means of numerous pixels is produced by acquiring and 
synthesizing a plurality of sets of frame information from the same video 
information. Since manual blurring may occur in video images acquired by 

20 digital video or the like, the deviation in the horizontal direction and 
vertical direction among images expressed by a plurality of sets of frame 
data is detected in pixel units, and the deviation in the vertical direction 
and horizontal direction is reduced by causing images superimposed on a 
reference image to perform a translational movement (parallel movement) 

25 by Nx pixels in the horizontal direction and Ny pixels in the vertical 
direction (Nx and Ny are positive integers). 
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[0003] Furthermore, the following method is disclosed in Japanese 
Patent Application Laid-Open No. 11-164264: specifically, in the case of a 
conversion in degree of resolution from images with a standard resolution to 
images with a high resolution, a memory for high resolution images is 
5 provided, the movement of the images is detected, and the respective pixels 
of the high resolution images are embedded using the pixels of a plurality of 
standard resolution images that are moved in parallel in accordance with 
the movement of the images. 

[0004] Manual blurring includes not only translational deviation in the 
10 horizontal direction and vertical direction, but also diagonal components. 
Consequently, manual blurring cannot be sufficiently corrected in the case of 
the abovementioned conventional techniques. Accordingly, there has been 
a desire to obtain still images with a high image quality from a plurality of 
sets of frame information in video information by more adequately 
15 correcting manual blurring. 

SUMMARY OF THE INVENTION 

[0005] The present invention was devised in light of the abovementioned 
problems; it is an object of the present invention to provide a technique that 
20 make it possible to obtain high-quality still images from a plurality of sets of 
frame information in video information. 

[0006] In the present invention, in order to achieve the abovementioned 
object, a plurality of sets of frame information are acquired from video 
information by a frame acquisition module. Then, on the basis of the 
25 plurality of sets of frame information acquired by the frame acquisition 
module, a deviation amount acquisition module acquires rotation amount 
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information that expresses the rotational deviation among images expressed 
by the same plurality of sets of frame information. Then, on the basis of 
rotation amount information acquired by the deviation amount acquisition 
module, a conversion module performs conversion processing that converts 
5 at least one set of the same plurality of sets of frame information so that at 
least one of the respective images expressed by the plurality of sets of frame 
information is rotated in a manner that reduces the rotational deviation 
among the images. Then, a synthesizing module synthesizes the plurality 
of sets of frame information on which conversion processing has been 

10 performed by the conversion module, so that image data that expresses 
images in halftones by means of numerous pixels is produced. 
[0007] Specifically, since the image data that expresses images in 
halftones is produced by synthesis from a plurality of sets of frame 
information in which the rotational deviation among the plurality of images 

15 expressed by the same plurality of sets of frame information has been 
reduced, manual blurring including rotational components is corrected. 
Accordingly, high-quality still images can be obtained from a plurality of 
sets of frame information in video information. 

For example, the abovementioned frame information or image data 

20 may conceivably comprise information in various types of data formats such 
as bit map data, compressed data or the like. Furthermore, the same 
frame information or image data may conceivably comprise various types of 
data such as YCbCr data constructed from brilliance data (Y) and 
chrominance difference data (Cb, Cr), RGB data constructed from red data 

25 (R), green data (G) and blue data (B), CMYaG data constructed from cyan 
data (C), magenta data (M), yellow data (Ye) and green data (G), Lab data in 
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the Lab space constituting the CIE standard or the like. 

[0008] The abovementioned numerous sets of frame information may 
constitute information that is continuous in a time series, and the 
abovementioned frame acquisition module may acquire a plurality of sets of 
5 frame information that are continuous in a time series from these numerous 
sets of frame information. Since image data that expresses still images is 
produced by the synthesis of frame information that is continuous in a time 
series with little variation, high-quality still images can be obtained using a 
simple construction. 

10 [0009] A construction may also be used in which the abovementioned 
deviation amount acquisition module acquires translational movement 
amount information expressing the translational deviation expressed by the 
abovementioned plurality of sets of frame information from these plurality 
of sets of frame information, and the abovementioned conversion module 

15 performs conversion processing on the basis of the abovementioned acquired 
translational movement amount information which converts at least one of 
the abovementioned plurality of sets of frame information so that at least 
one of the respective images expressed by the abovementioned plurality of 
sets of frame information is caused to undergo translational movement that 

20 reduces the translational deviation among the images. 

[00 10] Specifically, still images are produced by synthesizing frame 
information so that the translational deviation (vertical and horizontal 
deviation) among sets of frame information is reduced. Furthermore, the 
diagonal components of manual blurring are more reliably corrected by 

25 reducing the translational deviation of the specified central positions about 
which the images are rotated. Accordingly, even higher- quality still images 
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can be obtained. 

If the abovementioned conversion module performs conversion 
processing in which at least one of the abovementioned plurality of sets of 
frame information is converted so that the rotational deviation or 
5 translational deviation among the images is reduced by causing at least one 
of the respective images expressed by the abovementioned plurality of sets 
of frame information to rotate or perform a translational movement, an even 
more desirable construction is obtained. 

The abovementioned rotation amount information and translational 
10 movement amount information, as well as various types of information such 
as (for example) numerical value information, character information and the 
like, are conceivable. 

[0011] As a concrete example of the abovementioned conversion 

processing, the conversion module may perform conversion processing in 

15 which the positions of the respective pixels of at least one of the 
abovementioned plurality of sets of frame information are converted on the 
basis of the abovementioned rotation amount information and translational 
movement amount information so that both of the abovementioned types of 
image deviation are reduced. In this case, the conversion module may 

20 convert the positions of the respective pixels of at least one of the plurality 
of sets of frame information in units that are finer than these pixels. Since 
the position conversion (coordinate conversion) of the respective pixels of the 
frame information can be performed with high precision, even 
higher- quality still images can be obtained. 

25 [0012] The abovementioned deviation amount acquisition module may 
have a construction in which the abovementioned rotation amount 
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information and translational movement amount information are acquired 
by determining the rotational deviation and translational deviation among 
the abovementioned images on the basis of the positions of the 
abovementioned pixels and halftone data for these pixels in the 
5 abovementioned plurality of sets of frame information using a calculation 
system that determines this rotational deviation and translational deviation 
by the method of least squares. By using such a calculation system, it is 
possible to determine the rotational deviation and translational deviation at 
a high speed; accordingly, the speed of the processing that produces the 

10 image data can be increased. 

[0013] The abovementioned synthesizing module may have a 

construction in which the abovementioned image data is produced using 
numbers of pixels that are different from those of the abovementioned frame 
information. Since the numbers of pixels of the images expressed by the 

15 image data can be varied, the resolution of the still images (for example) can 
be varied, so that still images that are more in accordance with the images 
desired can be obtained. 

[0014] In this case, the abovementioned synthesizing module may also 
have a construction in which the abovementioned image data is produced 

20 using a number of pixels that is greater than that of the abovementioned 
frame information. Since the number of pixels of the images expressed by 
the image data is increased, (for example) still images with a high image 
quality can be produced with a higher resolution, so that still images that 
are more in accordance with the images desired can be obtained. 

25 Of course, even if the number of pixels of the images expressed by 

the image data is the same as the number of pixels of the images expressed 
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by the frame information, this merely module that the number of pixels of 
the still images cannot be varied; high-quality still images can still be 
obtained. 

[0015] In cases where image data is produced from three or more sets of 
5 frame information, the following construction may be used* specifically, the 
first set of rotation amount information that is acquired by the deviation 
amount acquisition module is information that expresses the rotational 
deviation of both images expressed by the first and second sets of frame 
information that are continuous in time series order. The second set of 

10 rotation amount information that is acquired by the deviation amount 
acquisition module is information that expresses the rotational deviation of 
both images expressed by the second and third sets of frame information 
that are continuous in time series order. The second set of frame 
information is converted by the conversion module so that the rotational 

15 deviation with respect to the image expressed by the first set of frame 
information is reduced. The third set of frame information is converted by 
the same conversion module so that the rotational deviation with respect to 
the image expressed by the first set of frame information is reduced. Here, 
since the rotational deviation among the images expressed by the second 

20 and third sets of frame information is generally less than the rotational 
deviation among the images expressed by the first and third sets of frame 
information, rotation amount information that expresses the rotational 
deviation with a higher degree of precision can be acquired. Accordingly, 
images with a higher image quality can be obtained. 

25 [0016] The abovementioned synthesizing module may also have a 
construction in which the abovementioned image data is produced by 
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performing specified interpolation processing from a plurality of sets of 
frame information on which the abovementioned conversion processing has 
been performed. Since image data for the respective pixels is produced by 
interpolation, still images with a higher image quality can be obtained. 
5 [0017] The abovementioned synthesizing module may also have a 
construction in which the abovementioned interpolation processing is 
performed using halftone data for pixels that are present in the areas 
surrounding a pixel of interest that produces the abovementioned image 
data (among the total pixels of the abovementioned plurality of sets of frame 

10 information) while this pixel of interest is successively moved, and the 
abovementioned image data is produced from the halftone data following 
interpolation processing. Since image data for the respective pixels is 
produced using halftone data for pixels present in the surrounding area, 
still images that have a higher image quality can be obtained. 

15 [0018] Here, even if the frame information is not converted so that 
rotational deviation or the like is reduced, an effect that causes the still 
images to have a higher image quality is obtained as a result of the 
abovementioned construction. Accordingly, the following construction may 
also be used. Specifically, the synthesizing module performs a specified 

20 interpolation using halftone data for pixels present in the area surrounding 
a pixel of interest that produces image data (among all of the pixels of a 
plurality of sets of frame information) while successively moving this pixel 
of interest, and image data is produced from the halftone data following this 
interpolation processing. Since image data for the respective pixels is 

25 produced using halftone data for pixels present in the surrounding area, 
still images that have a higher image quality can be obtained. 
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[0019] The abovementioned synthesizing module may also have a 
construction in which the abovementioned interpolation processing is 
performed using halftone data for the pixel that is closest to the 
abovementioned pixel of interest (among all the pixels of the 
5 abovementioned plurality of sets of frame information). When a test was 
conducted, no jaggy (zig-zag pattern) was seen in the edge portions of the 
image. Accordingly, still images with a higher image quality can be 
obtained using this construction. 

[0020] In this case, the abovementioned synthesizing module may also 
10 have a construction in which the abovementioned interpolation processing is 
performed using frame information that includes the pixel closest to the 
abovementioned pixel of interest (among the abovementioned plurality of 
sets of frame information). Still images with a high image quality can be 
obtained by means of a simple construction from a plurality of sets of frame 
15 information in video information! Especially in cases where the precision 
with which the movements of the individual frames is estimated is good, 
high-definition, high-resolution images can be obtained. 

[0021] Furthermore, the abovementioned synthesizing module may also 
have a construction in which the abovementioned interpolation processing is 

20 performed using frame information that includes pixels located within a 
specified range that uses the abovementioned pixel of interest as a reference 
(among the abovementioned plurality of sets of frame information). 
Consequently, in cases where the precision with which the movements of the 
individual frames is estimated is not good, the deviation of the images can 

25 be made inconspicuous, so that the image quality of the still images can be 
improved. 
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[0022] Here, the abovementioned synthesizing module may also perform 
the abovementioned interpolation processing by sets of frame information 
using frame information that includes pixels located within a specified 
range that uses the abovementioned pixel of interest as a reference (among 
5 the abovementioned plurality of sets of frame information), and may 
produce the abovementioned image data by summarizing the respective sets 
of halftone data following this interpolation processing. Accordingly, in 
cases where the precision with which the movements of the individual 
frames is estimated is not good, the image quality of the still images can be 

10 improved by means of a simple construction. 

Furthermore, various constructions such as arithmetical averaging, 
geometrical averaging, harmonic averaging (use of the reciprocal of the 
arithmetical mean of the reciprocals of the halftone data), averaging with 
different weighting for the respective sets of halftone data or the like are 

15 conceivable as constructions for the averaging of the respective sets of 
halftone data following the interpolation processing. 

[0023] Furthermore, in cases where pixels of the abovementioned frame 
information are not present within a specified range using the 
abovementioned pixel of interest as a reference, the abovementioned 

20 synthesizing module may perform the abovementioned interpolation 
processing using any of the abovementioned plurality of sets of frame 
information. Accordingly, image data can be reliably produced. 
[0024] Furthermore, the abovementioned synthesizing module may also 
have a construction n which the abovementioned interpolation processing is 

25 performed for each of the abovementioned plurality of sets of frame 
information, and the abovementioned image data is produced by 
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summarizing the respective sets of halftone data following this interpolation 
processing. Since pixels are present in the form of a lattice in each frame, 
still images with a high image quality can be obtained by utilizing 
interpolation processing with good precision such as processing performed 
5 by the by-cubic method or the like. 

Here, various constructions such as arithmetical averaging, 
geometrical averaging, harmonic averaging, averaging with different 
weighting for the respective sets of halftone data, summing or the like are 
conceivable as the construction used to summarize the respective sets of 

10 halftone data. 

[0025] The present invention can also be applied to an image deviation 
amount detection device. Specifically, the deviation amount acquisition 
module detects an amount of rotation that expresses the rotational 
deviation among images expressed by a plurality of sets of frame 

15 information on the basis of this frame information which is acquired by the 
frame acquisition module. Accordingly, if a plurality of sets of frame 
information are synthesized using the detected amount of rotation, still 
images with a high image quality can be obtained from a plurality of sets of 
frame information in video information. 

20 [0026] The abovementioned image producing device and image deviation 
amount detection device include various aspects as concepts of the present 
invention, and may be appropriately altered; e. g., these devices may be 
used independently in some cases, or may be used together with other 
methods in a state in which these devices are incorporated into certain types 

25 of equipment. 

Furthermore, the present invention is also found in the methods 
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used to perform processing corresponding to the abovementioned module in 
accordance with a specified procedure. Accordingly, the present invention 
can also be applied as a control method for an image producing device or 
image deviation amount detection device. 
5 Furthermore, the present invention can also be applied as a printing 

control device which performs printing control of a printer that performs 
printing on the basis of the produced image data, and can be further applied 
as a printing system comprising such as printing control device and printer; 
in such cases, the effect of the invention is basically the same. 

10 [0027] In the working of the present invention, there may also be cases 
in which specified programs are executed by the abovementioned devices. 
Accordingly, the present invention can also be applied as control programs 
for an image producing device or image deviation amount detection device. 
Furthermore, recording media which record such programs are commonly 

15 used, and it is conceivable that such programs may be appropriately read 
into a computer from such recording media. Specifically, the present 
invention can also be applied as a computer-readable recording medium on 
which such programs are recorded; in this case, the effect of the present 
invention is basically the same. 

20 Here, besides a magnetic recording medium or optical-magnetic 

recording medium, the abovementioned recording medium may be any type 
of recording medium that might be developed in the future. Furthermore, 
the stage of duplication may be any stage of duplication such as a primary 
duplicated product, secondary duplicated product or the like. Cases in 

25 which portions of the program are realized by hardware, and cases in which 
portions of the program are recorded on a recording medium and 
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appropriately read in as required, are also included in the concept of the 
present invention. 

These and other objects, features, aspects, and advantages of the 
present invention will become more apparent from the following detailed 
5 description of the preferred embodiments with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0028] Fig. 1 is a diagram which shows the schematic construction of the 
printing system; 

10 Fig. 2 is a block diagram which shows the construction of the 

printer; 

Fig. 3 is a diagram which shows the schematic construction of the 
image producing device in model form; 

Fig. 4 is a diagram which shows the construction of the frame 
15 information in model form; 

Fig. 5 is a diagram which shows the translational deviation amount 
and rotational deviation amount in model form; 

Fig. 6 is a diagram which shows the conditions of superimposition of 
the reference frame image and object frame images in model form; 
20 Fig. 7 is a diagram which shows the conditions of the estimation of 

the translational movement correction amount by the gradient method in 
model form! 

Fig. 8 is a diagram which shows the rotational correction amount of 
the pixels in model form; 
25 Fig. 9 is a diagram which shows the conditions of the estimation of 

the translational movement correction amount by the pattern matching 
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method in model form; 

Fig. 10 is a flow chart which shows the processing performed by the 
image producing device; 

Fig. 11 is a flow chart which shows the camera motion estimation 
5 processing; 

Fig. 12 is a flow chart which shows the frame conversion processing; 

Fig. 13 is a diagram which shows the conditions of coordinate 
conversion and superimposition of the object frame information in model 
form; 

10 Fig. 14 is a flow chart which shows the synthesis processing (part l); 

Fig. 15 is a diagram which shows the conditions of the selection of 
the closest pixel in model form; 

Fig. 16 is a diagram which shows the conditions of the interpolation 
processing performed by the by-linear method in model form; 
15 Fig. 17 is a flow chart which shows the synthesis processing (part 2); 

Fig. 18 is a diagram which shows the conditions of the selection of 
pixels within a specified range from the pixel of interest i in model form; 

Fig. 19 is a flow chart which shows the synthesis processing (part 3); 

and 

20 Fig. 20 is a diagram which shows the conditions of the processing 

performed by the image producing device in a modification in model form. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0029] Below, embodiments of the present invention will be described in 
25 the following order: 

(l) Construction of Printing System 
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(2) Outline of Construction of Image Producing Device and Image 
Deviation Amount Detection Device 

(3) Estimation of Camera Motion 

(4) Processing Performed by Image Producing Device and Image Deviation 
5 Amount Detection Device 

(4-1) Cutting Out of Synthesis Range 

(4-2) Detection of Translational Movement Correction Amount and 
Rotational Correction Amount 
(4-3) Frame Image Deviation Correction 
10 (4-4) Synthesis of Frame Images - Part 1 
(4-5) Synthesis of Frame Images — Part 2 
(4-6) Synthesis of Frame Images - Part 3 

(5) Embodiments 
[0030] 

15 (l) Construction of Printing System 

Fig. 1 shows the schematic construction of a printing system 100 
constituting one embodiment of the present invention. This system 100 is 
constructed from a personal computer (PC) 10, an ink jet printer 20 which is 
a printing device that is capable of color printing, and the like. 

20 Furthermore, the PC 10 constitutes the image producing device referred to 
in the present invention in the sense that this PC produces image data that 
expresses still images from a plurality of sets of frame information in video 
information, and likewise constitutes the image deviation amount detection 
device referred to in the present invention in the sense that this PC detects 

25 an amount of rotation that expresses the rotational deviation among images 
from the abovementioned plurality of sets of frame information. 
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Furthermore, this PC also constitutes a printing control device in the sense 
that this PC produces image data and causes the printer 20 to print 
corresponding still images. 

The PC 10 comprises a CPU 11 which constitutes the center of 
5 operational processing; this CPU 11 controls the PC 10 as a whole via a 
system bus 10a. A ROM 12, RAM 13, DVD-ROM drive 15, 1394 I/O 17a, 
various types of interfaces (I/F) 17b through 17e and the like are connected 
to this bus 10a. Furthermore, a hard disk (HD) 14 is also connected via a 
hard disk drive. A desktop PC is used as the computer in the present 

10 embodiment; however, any computer with a common construction can be 
used as this computer. 

An operating system (OS) and an application program (APL) which 
can create image information and the like are stored on the HD 14. At the 
time of execution, the CPU 11 sends this software to the RAM 13 as 

15 appropriate, and the program is executed while appropriately accessing the 
RAM 13 as a temporary work area. 

The 1394 I/O 17a is an I/O that conforms to the IEEE 1394 standard, 
and a digital video camera 30 or the like is connected to this I/O. This 
video camera 30 can produce video information and output this video 

20 information to the PC 10. A display 18a which displays images on the basis 
of frame information and image data is connected to the CRT I/F 17b, and a 
keyboard 18b and mouse 18c are connected to the input I/F 17c as operating 
input devices. Furthermore, the printer 20 is connected to the printer I/F 
17e via a parallel I/F cable. Of course, a construction in which the printer 

25 20 is connected via a USB cable or the like may also be used. 

[0031] Using C (cyan), M (magenta), Ye (yellow) and K (black) inks, the 
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printer 20 causes ink to jet onto printing paper (a printing medium) so that 
the amounts of ink correspond to the halftone values that form the image 
data; in this way, the printer 20 prints images. Of course, it would also be 
possible to employ a printer that uses inks other than inks of four colors. 
5 Furthermore, various types of printing devices such as laser printers, 
bubble type printers that cause the ink to jet by generating bubbles in the 
ink passages or the like may be employed. 

In the printer 20, as is shown in Fig. 2, a CPU 21, ROM 22, RAM 23, 
communications I/O 24, control IC 25, ASIC 26, I/F 27 and the like are 

10 connected via a bus 20a. The communications I/O 24 is connected to the 
printer I/F 17e of the PC 10, so that the printer receives printing jobs 
comprising data converted into CMYeK, a page description language and the 
like that is transmitted from the PC 10 via the communications I/O 24. 
The ASIC 26 outputs applied voltage data based on the CMYeK data to the 

15 head driving part 26a while transmitting and receiving specified signals to 
and from the CPU 21. This head driving part 26a produces an applied 
voltage pattern that is applied to piezoelectric elements contained in the 
printing head on the basis of this applied voltage data, and causes the 
printing head to jet CMYeK inks. The carriage mechanism 27a and paper 

20 feeding mechanism 27b connected to the I/F 27 cause the printing head to 
perform a main scan, and a sub -scan is performed by successively feeding 
out the medium while performing an appropriate page changing operation. 
Furthermore, the CPU 21 controls various parts in accordance with a 
program written into the ROM 22 while utilizing the RAM 23 as a work 

25 area. 

[0032] In the PC 10, BIOS is executed using the abovementioned 
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hardware as a foundation, and the OS and APL are executed on a higher 
level. Various types of drivers may be incorporated into the OS as printer 
drivers that control the printer I/F 17e or the like, and hardware control can 
be performed in this manner. Such printer drivers can perform two-way 
5 communications with the printer 20 via the printer I/F 17eJ these drivers 
receive image data from the APL, prepare printing jobs, and send these jobs 
out to the printer 20. The image production program and image deviation 
amount detection program of the present invention are constructed from 
APL, but may also be constructed by printer drivers. Furthermore, the HD 

10 14 is a medium on which these programs are recorded; however, this 
medium may also be (for example) a DVD-ROM 15a, CD-ROM, flexible disk, 
optical-magnetic disk, nonvolatile memory, punch cards or a printed 
medium on which symbols such as a bar code or the like are printed. Of 
course, the abovementioned programs stored in a specified server may also 

15 be downloaded from the communications I/F 17d via the internet, and then 
executed. 

Furthermore, the abovementioned hardware and the 
abovementioned programs may be used in conjunction to construct an image 
producing device and image deviation amount detection device. 
20 [0033] 

(2) Outline of Construction of Image Producing Device and Image 
Deviation Amount Detection Device 

Fig. 3 shows the schematic construction of the abovementioned 
image producing device in model form. The image production program that 
25 causes the PC to function as an image producing device U0 is constructed 
from a plurality of modules corresponding to the respective types of module 
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described below. 

The video information Dl is constructed from numerous sets of 
frame information D2 that express frame images in halftones by means of 
numerous pixels in the form of a dot matrix. These sets of frame 
5 information D2 constitute information that is continuous in a time series. 
In the figures, the order of the time series is shown from left to right, i. e., 
information is shown as later information in time series order as this 
information is located further toward the right side. The frame acquisition 
module Ul acquires a plurality of sets of frame information D3 that is 

10 continuous in a time series from the video information Dl. In the present 
embodiment, the first set of frame information in time series order is 
designated as reference frame information D31, and the sets of frame 
information that follow this reference frame information (three sets in the 
example shown in the figures) are designated as object frame information 

15 D32. Of course, any set of frame information among the plurality of sets of 
frame information may be used as reference frame information. The 
reference frame information need not be information of the acquired 
plurality of sets of frame information. 

[0034] Fig. 4 shows the construction of the frame information D2 in 
20 model form. As is shown in the figure, each set of frame information D2 
expresses a frame image by halftone data D8 for numerous pixels 51 in the 
form of a dot matrix. The halftone data D8 in the present embodiment is 
described as YCbCr data comprising Y (brilliance), Cb (blue chrominance 
difference) and Cr (red chrominance difference); however, this data may 
25 also be RGB data comprising R (red), G (green) and B (blue) or the like. 
Furthermore, the respective YCbCr components are described as comprising 
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256 gradations; however, these components may also have 1024 gradations, 
100 gradations or the like. 

[0035] On the basis of the plurality of sets of frame information D3, the 
deviation amount acquisition module U2 detects the amount of rotational 
5 correction and amount of translational movement correction which are 
correction amounts that are used to correct the amount of rotational 
deviation that expresses the rotational deviation among the images 
expressed by the plurality of sets of frame information, and the amount of 
translational movement deviation that expresses the translational 

10 movement deviation among these images (this module correspond to the 
deviation amount detection module referred to in the present invention), 
and thus acquire rotation amount information D4 and translational 
movement amount information D5. In the present embodiment, this 
module is arranged so that the amount of rotational correction and amount 

15 of translational movement correction are detected for each of the sets of 
object frame information D32 following the initial reference frame 
information D31 in time series order; however, the rotational correction 
amount and translational movement correction amount can be detected in 
various ways in regard to which of the sets of frame information is used as 

20 reference frame information and the like. 

Here, the relationship between the amount of deviation and the 
amount of correction will be described with reference to Fig. 5. As is shown 
in Fig. 5, numerous pixels of the frame information are expressed on the xy 
plane; the upper sets of frame information 52a and 52b are arranged in time 

25 series order, and the lower sets of frame information 52c and 52d are also 
arranged in time series order. As is shown in the upper part of the figure, 
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the amount of movement ua (= 1 pixel) in the direction of the x axis and the 
amount of movement va (= 2 pixels) in the direction of the y axis when the 
object 53a on the frame information 52a is caused to undergo parallel 
movement without being rotated are translational movement deviation 
5 amounts. Furthermore, as is shown in the lower part of the figure, the 
amount of movement when the object 53c on the frame information 52c 
rotates about the center 52cl of the frame image 52c in a case where the 
translational movement deviation amounts (ua, va) are (0, 0) is a rotational 
deviation amount (8a). In this case, in regard to the amounts of correction 

10 that are used to eliminate these deviations, if the amount of rotational 
correction is designated as "6" and the amounts of translational movement 
correction are designated as "u" in the case of the horizontal direction and 
"v" in the case of the vertical direction, then the relationship between the 
amount of rotational deviation and amount of rotational correction can be 

15 expressed as 5 = —5a , and the relationships between the amounts of 
translational movement deviation and amounts of translational movement 
correction can be expressed as u = — ua and v = -va. Furthermore, the 
amount of rotational correction 8 constitutes the rotation amount 
information referred to in the present invention, and the amounts of 

20 translational movement correction (u, v) constitute the translational 
movement amount information referred to in the present invention. 

The deviation amount acquisition module U2 of the present 
embodiment is devised so that this module determines, on the basis of the 
positions of the pixels of the plurality of sets of frame information D3 and 

25 the halftone data for these pixels, the rotational deviation and translational 
movement deviation among images expressed by the plurality of sets of 
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frame information using calculation formulae for determining this 
rotational deviation and translational movement deviation by the method of 
least squares, and thus acquire rotation amount information D4 and 
translational movement amount information D5. The procedure used to 
5 determine this rotational deviation and translational movement deviation is 
called "estimation of camera motion". 

[0036] On the basis of the rotation amount information D4 and 

translational movement amount information D5, the conversion module U3 
performs conversion processing in which at least one of the plurality of sets 

10 of frame information D3 is converted by causing rotation or translational 
movement of at least one of the respective images expressed by the plurality 
of sets of frame information D3 so that the rotational deviation or 
translational movement deviation among the images is reduced. The 
conversion module U3 of the present embodiment converts at least one of 

15 the plurality of sets of frame information D3 (in concrete terms, the object 
frame information D32) so that the deviation among the images is 
eliminated, and thus produces a plurality of sets of frame information D6 
that have been subjected to this conversion processing. In this case, the 
positions of the respective pixels of at least one of the plurality of sets of 

20 frame information are converted in units that are finer than these pixels. 

Here, in cases where there is a second frame image following the 
first frame image, if the amounts of translational movement deviation from 
the first frame image to the second frame image are designated as (ua, va) 
and the amount of rotational deviation is designated as (5a), then, in a case 

25 where the deviation of the second frame image with respect to the first 
frame image is to be eliminated, the second frame image is caused to 
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undergo translational movements of (— ua, -va) and a rotational movement 
of (—5a). Furthermore, in a case where the deviation of the first frame 
image with respect to the second frame image is to be eliminated, this can 
be accomplished by causing the first frame image to undergo translational 
5 movements of (ua, va) and a rotational movement of (5a). Of course, it 
would also be possible to cause the first frame image to undergo 
translational movements of (ua/2, va/2) and a rotational movement of (5a/2), 
and to cause the second frame image to undergo translational movements of 
(-ua/2, -va/2) and a rotational movement of (^5a/2). 

10 [0037] The synthesizing module U4 produces image data D7 by 

synthesizing the plurality of sets of frame information D6 on which 
conversion processing has been performed. This image data is data that 
expresses still images in halftones by means of numerous pixels in the form 
of a dot matrix. The image data D7 in the present embodiment is described 

15 as RGB data with 256 RGB gradations; however, this data may also be 
YCbCr data or the like, and the number of gradations may be 1024 
gradations, 100 gradations or the like. The image data in the present 
embodiment is described as data in which the number of pixels is different 
from that of the frame information; the number of pixels is increased 

20 compared to that of the frame information, so that the resolution of the data 
is increased. However, the image data that is produced may have the same 
number of pixels as the frame information, or may have a smaller number of 
pixels than the frame information. 

When a plurality of sets of frame information are synthesized, the 

25 halftone value of the pixel of interest that produces the image data is 
determined by performing specified interpolation processing using the 
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halftone data for pixels that are present in the area surrounding the pixel of 
interest (among all of the pixels of the plurality of sets of frame information) 
while successively moving this pixel of interest, so that image data D7 that 
expresses still images in halftone values by individual pixels is produced. 
5 [0038] In conventional techniques, only the translational movement 
deviation in the horizontal direction and vertical direction among frame 
images is corrected when still images are synthesized from a plurality of 
frame images; accordingly, manual blurring which includes diagonal 
components as well cannot be sufficiently corrected. Since the still images 

10 expressed by the image data produced by the present image producing 
device UO are produced by synthesis from a plurality of frame images with 
the rotational deviation eliminated along with the translational movement 
deviation among the plurality of frame images, manual blurring including 
diagonal components is corrected. Accordingly, still images with a high 

15 image quality can be obtained from a plurality of sets of frame information 
in video information, and one scene from a moving image can be expressed 
as a high-resolution still image, so that the definition can be increased in 
the case of printing. 
[0039] 

20 (3) Estimation of Camera Motion 

Next, the estimation of the camera motion that is performed by the 
deviation amount acquisition module and deviation amount detection 
module will be described. The "estimation of the camera motion" refers to a 
process that estimates the relative positional relationship of a plurality of 

25 frame images expi*essed by a plurality of sets of frame information cut out 
from the video information. Here, it is assumed that there is no movement 
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of the object of imaging itself or the set position of the video camera during 
imaging, so that only the orientation of the video camera varies. 
Specifically, this corresponds to camera work called "panning" and "tilting". 
Furthermore, in order to describe this in a clearly understandable manner, 
it is assumed that there are object frame images expressed by object frame 
information following the reference frame image expressed by reference 
frame information, and it is assumed that the amount of deviation between 
the reference frame image and object frame images is estimated. 
[0040] It is assumed that the coordinates (xl, yl) of the reference frame 
information correspond to the coordinates (x2, y2) of the object frame 
information, as shown in Fig. 6. The translational movement correction 
amounts are designated as (u, v), and the rotational correction amount is 
designated as 8 with the center of the frame image as the origin. Since it is 
assumed that the focal distance is not varied during imaging, the following 
equations are used as coordinate conversion equations, with conversion of 
only translational movement and rotation being taken as a prerequisite. 

x2 = cos 8 • xl + cos 8 • yl - u ...(l) 

y2 = - sin 8 • xl + cos 8 • yl - v .. ; (2) 

xl = cos 8 • (x2 + u) - sin 8 • (y2 + v) . . .(la) 

yl = sin 8 • (x2 + u) + cos 8 - (y2 + v) .(2a) 
Furthermore, since the time difference between the reference frame image 
and the object frame images is extremely small, u, v and 8 are extremely 
small quantities. Here, since cos 8 « 1 and sin 8 « 8 when 8 is an extremely 
small quantity, the above equations can be replaced by the following 
equations" 

x2 = xl + 8 yl-u ...(3) 
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y2 = -8-xl + yl-v ...(4) 

xl = (x2 + u) - 8 - (y2 + v) ...(3a) 

yl = 5 • (x2 + u) + (y2 + v) ...(4a) 

Furthermore, u, v and 8 in Equations (3) and (4) (or (3a) and (4a)) 
5 can be estimated by the method of least squares. 

[0041] The estimation of the camera motion is based on the gradient 
method, in which the positions of the pixels are estimate in units finer than 
1 pixel using (for example) the brilliance of the respective pixels among the 
sets of frame information. 

10 Here, as is shown in the upper part of Fig. 7, the brilliance values of 

the respective pixels of the reference frame information are expressed as 
zl(ix, iy), and the brilliance values of the respective pixels of the object 
frame information are expressed as z2(ix\ iyO. 

First, assuming that the coordinates (ix\ iy') of the object frame 

15 information are between the coordinates (ix ~ ix + 1, iy ~ iy + l) of the 
reference frame information, the procedure used to determine the 
coordinates (ix', iy') by the gradient method will be described. 
[0042] As is shown in the middle part of the figure, if the position of the 
coordinates (ix', iy') in the direction of the x axis is designated as ix + Ax, 

20 and it is assumed that Px = zl(ix +1, iy) — zl(ix, iy), then it is sufficient to 
determine a value for Ax which is such that Px x Ax = z2(ix', iy') - zl(ix, iy). 
In actuality, Ax is determined for each pixel, and is average overall. Here, 
if the simple expressions of zl = zl(ix, iy) and z2 = z2(ix', iy') are used, then 
it is sufficient to calculate a value of Ax which is such that the following 

25 equation holds true- 

{Px • Ax - (z2 - zl)} 2 = O ...(5) 
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Furthermore, as is shown in the bottom part of the figure, it the 
position of the coordinates (ix\ iy') in the direction of the y axis is designated 
as iy + Ay, and it is assumed that Py = zl(ix, iy + 1) — zl(ix, iy), then it is 
sufficient to determine a value for Ay which is such that Py • Ay = z2(ix\ iyO 
5 - zl(ix, iy). Here as well, if the simple expressions of zl = zl(ix, iy) and z2 
= z2(ix\ iyO are used, then it is sufficient to calculate a value of Ay which is 
such that the following equation holds true: 

{Py • Ay.- (z2 - zl)} 2 = O ...(6) 

Accordingly, if both the x and y directions are considered, it is 
10 sufficient to determine Ax and Ay which minimize S 2 in the following 
equation by the method of least squares. 

S 2 = I {Px • Ax + Py • Ay - (z2 - zl)} 2 ...(7) 
[0043] Above, a procedure in which the amount of translational 

movement correction was determined by the gradient method based on the 
15 assumption of parallel movement by the frame images in the direction of the 
x axis and direction of the y axis was described. In the present invention, 
the rotation of the frame images is also taken into account. The procedure 
used in this case will be described below. 

As is shown in Fig. 8, if the distance of the coordinates (x, y) of the 
20 reference frame information from the origin O is designated as r, and the 
angle of rotation from the x axis is designated as 9, then r and 0 can be 
determined by the following equations " 

r = ( x 2 + y2)l/2 ..,(8) 

9 = tan" 1 (x/y) ...(9) 

25 

[0044] Here, assuming that the translational movement deviation has 
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been corrected, if the origins of the reference frame image and object frame 
image are aligned, and a rotation 8 is performed from the coordinates (x, y) 
so that the coordinates (x\ y') are obtained in the case of the object frame 
image, the amount of movement in the direction of the x axis and the 
amount of movement in the direction of the y axis caused by this rotation 
can be determined by the following equations" 

x - x * - r 5sin 9 = - 8 • y ...(10) 

y' - y » r 8cos 9 = 8 • x ...(11) 

Accordingly, Ax and Ay in the abovementioned Equation (7) can be 
expressed by the following equations using the translational movement 
correction amounts u and v and the rotational correction amount 8- 

Ax = u-8y ...(12) 

Ay = v + 8-x ...(13) 

If these equations are substituted into the abovementioned Equation 
(7), the following equation is obtained* 

S 2 = I {Px • (u - 5 • y) + Py • (v + 8 • x) - (z2 - zl)} 2 ...(14) 
[0045] Specifically, if the coordinates of the reference frame information 
are designated as (ix, iy), and the coordinate values and halftone data 
(brilliance values) for all of the pixels of the reference frame information are 
substituted into Equation (14), then it is sufficient to determine u, v and 8 
that minimize S 2 by the method of least squares. The following equations 
can be obtained by the method of least squares* 
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u = {(m 8 *M02 — m v 2 )c u H-(m u *m v — m 6 *Mll)c v 

+ (m v •Mll-m u -M02)c 6 } / d ••• (15) 

v = {(m u *m v — nig *Mll)c u + (ma *M20— m u 2 )c v 

+ (m u -Mll-mv -M20)c 6 } / d — (16) 

6 = {(m v -Mll— m u -M02)c u + (m u 'Mil— m v -M20)c v 

+ (M20-M02-Mll 2 )c 6 }/d —(17) 

Here, i is a number distinguishing all of the pixels, and a, p, D, Nl through 
N5 and M are determined as shown below. 
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[0046] Accordingly, the translational movement correction amounts (u, 
v) and rotational correction amount (8) can be detected according to the 
camera motion using the abovementioned Equations (15) through (29), and 
these amounts can be acquired as translational movement amount 
information and rotation amount information. Here, since the 
translational movement correction amounts are designated in units smaller 
than the pixels of the frame images, detection can be performed with good 
precision. Furthermore, if a plurality of sets of frame information are 
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synthesized using the detected translational movement correction amounts 
and rotational correction amount, then still images with a high image 
quality can be obtained from a plurality of sets of frame information in the 
video information. 

5 [0047] Furthermore, the system may also be devised so that the 

translational movement correction amounts are detected in pixel units only 
for the translational movement deviation prior to the abovementioned 
estimation, and so that the translational movement deviation among the 
frame images is corrected in pixel units (i. e., roughly corrected) beforehand. 

10 If this is done, the translational movement amount information and rotation 
amount information can be acquired with even better precision, and still 
images with an even higher image quality can be obtained by synthesizing a 
plurality of frame images using this translational movement amount 
information and rotation amount information. 

15 An estimation procedure using the pattern matching method is 

known as a procedure for roughly detecting the translational movement 
deviation among frame images. 

[0048] Fig. 9 shows the conditions of estimation of the translational 
movement correction amount by the pattern matching method in model 
20 form. 

As is shown in the upper part of the figure, the brilliance values of 
the respective pixels of the reference frame information are expressed as 
zl(ix, iy), and the brilliance values of the pixels in the same positions in the 
object frame information are expressed as z2(ix, iy). 
25 First, using a case in which the pixel (ix, iy) of the reference frame 

information is caused to correspond to the pixel (ix, iy) of the object frame 



31 



PF04J801 



information as a reference, positions which are such that L shown below is 
minimized are determined for this case and for cases in which the 
abovementioned pixel is caused to correspond to the pixels (ix + 1, iy), (ix - 1, 
iy), (ix, iy + l) and (ix, iy -l) of the object frame information. 
5 L= | Z2i - Zli | ...(30) 

or 

L=(Z2i-Zli) 2 ...(31) 
[0049] For example, when L of the pixel (ix —1, iy) of the object frame 
information shows a minimum value, positions in which L shows a 

10 minimum value are determined for a case in which the pixel (ix, iy) of the 
reference frame information is caused to correspond to the pixel (ix — 1, iy) 
of the object frame information, and for cases where this pixel is caused to 
correspond to the pixels (ix, iy), (ix -2, iy), (ix — 1, iy + l) and (ix —1, iy — l) of 
the object frame information. Thus, a search for pixels of the object frame 

15 information where L shows a minimum value is repeated. 

Furthermore, for example, when L of the pixel (ix - 3, iy - l) of the 
object frame information shows a minimum value, positions in which L 
shows a minimum value are determined for a case in which the pixel (ix, iy) 
of the reference frame information is caused to correspond to the pixel (ix — 

20 3, iy - 1) of the object frame information, and for cases in which this pixel is 
caused to correspond to the pixels (ix - 2, iy — l), (ix — 4, iy — 1), (ix — 3, iy) 
and (ix - 3, iy -2) of the object frame information. When L of the center 
pixel (ix — 3, iy — 1) of the search shows a minimum value, the search is 
ended here. The shift of the object frame information at the time of 

25 completion of the search is (3, 1). 

[0050] Generally, in cases where the position of the pixel of the object 
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frame information that is caused to correspond to the pixel (ix, iy) of the 
reference frame information at the time of completion of the search is (ix - 
Ax, iy — Ay), the amount of translational movement correction from the 
reference frame image to the object frame image can be expressed as (Ax, 
5 Ay) in pixel units. 

Accordingly, if the pixels are shifted by the translational movement 
correction amount (Ax, Ay) for the object frame information, the estimation 
of the camera motion can be performed with a high degree of precision. 
[0051] 

10 (4) Processing Performed by the Image Producing Device and Image 

Deviation Amount Detection Device 

Below, the operation of the image producing device and image 

deviation amount detection device will be described in detail along with the 

processing performed by the image producing device and image deviation 
15 amount detection device. 

Fig. 10 uses a flow chart to show the processing performed by this 

image producing device. In concrete terms, the CPU 11 of the PC 10 

performs this processing. 

[0052] 

20 (41) Cutting Out of Synthesis Range 

First, a plurality of sets of frame information in the range that is to 
be synthesized are acquired by being cut out by the frame acquisition 
module from video information comprising numerous sets of frame 
information expressing images by YCbCr data (halftone data) for numerous 

25 individual pixels (step S105; "step" is hereafter omitted). There are no 
particular restrictions on the frame information that is cut out, as long as 
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this information comprises two or more sets of frame information. In the 
present embodiment, the system is devised so that four sets of frame 
information that are continuous in a time series are acquired from the video 
information. However, even if a plurality of discrete sets of frame 
5 information that are not continuous in a time series are acquired, the 
translational movement correction amount and rotational correction amount 
can be detected, and frame information can be synthesized so that still 
images are obtained. For example, in cases where the video information is 
constructed from independent frame information which has halftone data 

10 for all of the pixels and which can express images without reference to other 
frame information, and non-independent frame information which does not 
have halftone data for all of the pixels and which cannot express images 
without reference to other frame information, the following camera motion 
estimation processing can be performed at a high speed by referring only to 

15 the discrete independent frame information. 

[0053] Various positions are conceivable as the positions where frame 
information is cut out. For example, as is shown in Fig. 3, in cases where a 
specified key operation or button input operation is received while the 
playback of video from the video information is stopped, frame information 

20 that expresses the frame image that is temporarily stopped and displayed 
on the display (reference frame information D3l), and a specified number of 
sets of frame information (object frame information D32) that follow the 
abovementioned frame information in time series order, can be acquired as 
cut-out frame information. Of course, frame information expressing a 

25 frame information that is temporarily stopped as displayed on the display, 
and a specified number of sets of frame information that precede the 
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abovementioned frame information in time series order, may also be 

acquired as cut-out frame information. 

[0054] 

(4*2) Detection of Translational Movement Correction Amount and 
5 Rotational Correction Amount 

Next, camera motion estimation processing which estimates the 
translational movement correction amount and rotational correction amount 
according to the camera motion and acquires translational movement 
amount information and rotation amount information is performed by the 
10 deviation amount acquisition module and deviation amount detection 
module (S110). 

Fig. 11 shows the camera motion estimation processing by means of 
a flow chart. 

First, reference frame information is set from a plurality of sets of 
15 frame information acquired from the video information (S205). In the 
present embodiment, the first set of frame information in time series order 
is set as the reference frame information. Here, the frame information 
used as reference frame information may be set by storing this frame 
information in a specified region of the RAM, or this frame information used 
20 as reference frame information may be set by storing the memory address of 
this frame information in a specified pointer in the RAM. 

[0055] Next, the object frame information is set from the plurality of sets 
of frame information acquired from the video information (S210). In the 
present embodiment, since four sets of frame information are acquired, the 
25 object frame information is set from the frame information that remains 
after the reference frame information has been removed from these four sets 
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of frame information. For example, the second set of frame information in 
time series order is set as the object frame information. Here as well, the 
frame information that is used as object frame information may be stored in 
a specified region of the RAM, or the memory address of the frame 
5 information used as object frame information may be stored in a specified 
pointer. 

[0056] Furthermore, the respective variables (parameters) used in the 
abovementioned Equations (15) through (29) which are used to calculate the 
translational movement correction amounts u and v that correct the 

10 translational movement deviation and the rotational correction amount 5 
that corrects the rotational deviation by the method of least squares are 
initially set (S215). For example, 0 is substituted for the variables M20, 
Mil, M02, m u , m v , m s , c u , c v and ca that are used for sum calculations. 

Afterward, the position of the pixel of interest i is set from the 

15 reference frame information, and the position of the pixel of interest i' is set 
from the object frame information (S220). For example, in a case where the 
frame information is constructed in the form of a dot matrix comprising nx 
pixels in the horizontal direction (direction of the x axis) and ny pixels in the 
vertical direction (direction of the y axis), the position of the pixel of interest 

20 i can be set by i = ix + nx x iy when a pixel of coordinates (ix, iy) in the 
reference frame information is set as the pixel of interest, and the position of 
the pixel of interest i' can be set by i' = ix' + nx x iy' when a pixel of 
coordinates (ix', iy') in the object frame information is set as the pixel of 
interest. In the present embodiment, the order in which the positions of 

25 the pixels of interest are set begins from the pixel at the upper left, proceeds 
in order to the pixel at the upper right, and subsequently proceeds in order 
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one at a time from the pixel at the left end below to the pixel at the right 
end, finally ending at the lower right pixel. The same is true when the 
positions of pixels of interest are set in various types of processing below. 
Of course, the order in which the positions of the pixels of interest are set 
5 may be appropriately altered, and different orders may be used in 
accordance with the type of frame images involved or the like. 

Furthermore, a rough translational movement correction amount 
(Ax, Ay) from the reference frame image to the object frame image may be 
detected and acquired by the abovementioned pattern matching method, 
10 and the pixels may be subjected to a translational movement correction (Ax, 
Ay), prior to the processing of step S220. 

[0057] When the positions of the pixels of interest i and i' are set, the 
values of Pti, ki, M20, Mil, M02, m u , m v , m 5 , c u , c v and c§ used in the 
abovementioned Equations (18) through (28) are successively calculated 
15 (S225). 

For example, in the case of M20, it is necessary to calculate a sum! 
accordingly, for the variable M20, processing that performs a calculation of 

M20 <r- M20 + (values of pixels of interest i and i' within Z), 
i. e., processing that adds the values of the pixels of interest i and i' within Z, 
20 is performed. The same is true regarding Mil, M02, m u , m v , ms, c u , c v and 
c s . 

Then, a judgment is made as to whether or not the abovementioned 
variables have been calculated for all of the pixels of the reference frame 
information (S230). In cases where pixels for which the abovementioned 
25 variables have not been calculated remain, the processing of S220 through 
S230 is repeated while successively moving the pixels of interest i, i' for 
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which the abovementioned variables are calculated. Then, when the 
abovementioned variables have been calculated for all of the pixels, the 
processing proceeds to S235. 

[0058] In S235, d is calculated using the abovementioned Equation (29). 
5 Next, the translational movement correction amounts u and v are 

calculated using the abovementioned Equations (15) and (16) (S240). 
Specifically, the translational movement correction amounts that correct the 
translational movement deviation among the images expressed by the 
plurality of sets of frame information are detected in units finer than 1 pixel, 
10 and translational movement amount information is acquired as 
translational movement correction amounts u, v that constitute numerical 
information. 

Furthermore, the rotational correction amount 8 is calculated using 
the abovementioned Equation (17). Specifically, the rotational correction 
15 amounts that correct the rotational deviation among the images expressed 
by the plurality of sets of frame information are detected in units finer than 
1 pixel, and rotation amount information is acquired as a rotational 
correction amount 8 that constitutes numerical information. 

Then, u, v and 8 are stored in a specified region of the RAM as 
20 translational movement amount information and rotation amount 
information (S250). 

[0059] Afterward, a judgment is made as to whether or not u, v and 8 
have been acquired for all of the frame information (S255). In cases where 
there is remaining frame information for which u, v and 8 have not been 
25 acquired, the processing of S205 through S255 is repeated. In the present 
embodiment, the first set of frame information in time series order is always 
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set as the reference frame information, and the object frame information is 
set in time series order from the remaining frame information; then, the 
abovementioned translational movement amount information and rotation 
amount information are acquired using this frame information. When u, v 
5 and 5 have been acquired for all of the frame information, the present flow 
is ended. 

In the above processing the translational movement correction 
amounts (u, v) and rotational correction amount (5) are detected in 
accordance with the camera motion, and these data can be acquired as 

10 translational movement amount information and rotation amount 
information. In this case, since the conversion of the positions of the 
respective pixels of the frame information is performed with high precision 
in units finer than these pixels, still images with a high image quality can 
be obtained. Furthermore, since calculation formulae that correct the 

15 rotational deviation and translational movement deviation among the frame 
images by the method of least squares are used, the rotational correction 
amount and translational movement correction amounts can be detected at 
a high speed, so that the speed of the processing that produces the image 
data can be increased. 

20 Furthermore, translational movement amount information and 

rotation amount information may also be obtained by multiplying u, v and 5 
by specified correction coefficients in accordance with the properties of the 
frame images or the like. Even if these correction coefficients are not 1, the 
translational movement amount information is information that can reduce 

25 the translational movement deviation among the images expressed by a 
plurality of sets of frame information, and the rotation amount information 
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is information that can reduce the rotational deviation among the images 

expressed by a plurality of sets of frame information. 

[0060] 

(4-3) Frame Image Deviation Correction 
5 When the camera motion estimation processing is completed, the 

processing proceeds to S115 in Fig. 10, and conversion processing which 
converts the three sets of object frame information by causing the three 
object frame images expressed by the object frame information to rotate and 
undergo translational movement so that the translational movement 
10 deviation and rotational deviation with respect to the reference frame image 
is reduced is performed on the basis of the abovementioned translational 
movement amount information and rotation amount information by the 
conversion module. 

Fig. 12 shows the frame conversion processing by means of a flow 

15 chart. 

First, the object frame information for which the coordinates of the 
pixels are to be converted are set from the object frame information (S305). 
In the present embodiment, the object frame information is set in time 
series order in the same manner as in S210 in Fig. 11. 
20 Next, the position of the pixel of interest i' whose coordinates are to 

be converted is set from the object frame information (S310). In the 
present embodiment, the position of the pixel of interest i' is set in the same 
manner as in S220 in Fig. 11. 

[0061] Subsequently, the translational movement correction amounts u 
25 and v are read out from the RAM, and the position of the pixel of interest i' 
is subjected to a coordinate conversion so that this pixel is caused to 
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undergo a translational movement of u in the direction of the x axis and a 
translational movement of v in the direction of the y axis, thus canceling the 
translational movement deviation of the object frame image (S315). 
Specifically, the position of the pixel of interest i' with coordinates of (ix', iy') 
5 is caused to undergo a parallel movement to the coordinates (ix' + u, iy' + v). 
Here, the coordinates (ix' + u, iy' + v) are analog quantities, and are thus 
made finer than the pixels of the frame information. 

Furthermore, the rotational correction amount 8 is read out from the 
RAM, and a coordinate conversion is performed so that the position of the 

10 pixel of interest i is further caused to undergo a rotation of 8 with the center 
of the frame image as the origin, thus canceling the rotational deviation of 
the object frame image (S320). Here, if the coordinates (ix' + u, iy' + v) are 
replaced by the coordinates (x\ y') centered on the origin, and the distance 
from the origin to the coordinates (x\ y') is designated as r', then the 

15 coordinates (x', y') are caused to move by - 8 x y' in the direction of the x axis 
and 8 • x' in the direction of the y axis by a rotation of 8. Specifically, the 
position of the pixel of interest i' with coordinates of (ix' + u, iy' + v) following 
translational movement is moved to the coordinates of (x' — 8 • y\ y' + 8 • x'). 
Of course, the coordinates (x' - 8 • y\ y' + 8 • x') are finer than the pixels of 

20 the frame information. 

[0062] Subsequently, a judgment is made as to whether or not coordinate 
conversion has been performed for all of the pixels of the object frame 
information (S325). In cases where pixels remain for which coordinate 
conversion has not been performed, the processing of S305 through S325 is 

25 repeated while successively moving the pixel of interest i'; on the other hand, 
in cases where it is judged that coordinate conversion has been performed 
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for all of the pixels, the processing proceeds to S330. 

In S330, a judgment is made as to whether or not coordinate 
conversion has been performed for all of the sets of object frame information. 
In cases where object frame information remains for which coordinate 
5 conversion has not been performed, the processing of S305 through S330 is 
repeated. In the present embodiment, the object frame information is set 
in time series order, and the abovementioned coordinate conversion is 
performed. On the other hand, in cases where it is judged that coordinate 
conversion has been performed for all of the sets of object frame information, 

10 the present flow is ended. 

As a result of the above processing, the object frame information can 
be converted by causing the object frame information to undergo a 
translational movement and rotational movement on the basis of the 
translational movement amount information and rotation amount 

15 information so that rotational deviation between the images expressed by 
the reference frame information and object frame information is eliminated. 

Furthermore, it would also be possible to perform a coordinate 
conversion after multiplying u, v and 5 by specified correction coefficients in 
accordance with the properties of the frame images or the like. Even if the 

20 correction coefficients are not 1, the frame information can be converted so 
that the translational movement deviation and rotational deviation among 
images expressed by a plurality of sets of frame information is reduced. 

Furthermore, it would also be possible to perform a coordinate 
conversion on two sets of frame information using u, v and 8 multiplied by 

25 specified correction coefficients. 
[0063] 
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(4-4) Synthesis of Frame Images - Part 1 

When the frame conversion processing is completed, the processing 
proceeds to S120 in Fig. 10, and processing in which image data that 
expresses still images in halftones by means of numerous pixels is produced 
5 by the synthesis of the reference frame information and the three sets of 
object frame information for which conversion processing has been 
performed by the abovementioned frame conversion processing is performed 
by the synthesizing module. 

As is shown in Fig. 13, processing is performed in which the three 

10 sets of object frame information that have been subjected to coordinate 
conversion are superimposed in accordance with the estimation of the 
camera motion from the four sets of frame information so that the 
translational movement deviation and rotational deviation with respect to 
the reference frame information is eliminated. 

15 [0064] Fig. 14 shows the synthesis processing by means of a flow chart. 

First, the resolution of the still images to be expressed by the image 
data that is produced is acquired (S405). For example, a printing interface 
screen which has a resolution selection column can be displayed on the 
display, and a parameter that expresses the resolution of the still images 

20 following synthesis can be acquired from operating input into this resolution 
selection column, and can be stored in the RAM. Then, the ratio of the 
resolution of the still images to the resolution of the frame images 
constitutes the enlargement rate of the images, and the number of pixels of 
the image data following synthesis is determined so that this enlargement 

25 rate is obtained (S410). Thus, since the resolution of the still images can be 
varied and the number of pixels can be varied, still images desired by the 
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user can be obtained. Since the image producing device of the present 
invention makes it possible to produce image data that expresses still 
images with a high image quality by synthesizing a plurality of sets of frame 
information, still images with a high image quality in which the number of 
5 pixels is increased can be obtained by setting the still images at a higher 
resolution than the frame images, so that more desirable still images can be 
obtained. 

Below, the production of image data that expresses still images will 
be described using a case in which the pixels of the frame images are 

10 increased by 1.5 times in both the direction of the x axis and the direction of 
the y axis as an example. This image producing device produces image 
data by performing specified interpolation processing using the halftone 
data of pixels that are present in the area surrounding the pixel of interest 
that produces the image data (among all the pixels of the reference frame 

15 information and object frame information) while successively moving this 
pixel of interest. 

[0065] Subsequently, the position of the pixel of interest i that produces 
the image data that expresses still images in halftones is set (S415). In the 
present embodiment, the position of the pixel of interest i corresponding to 
20 the still image is set in the same manner as in S220 in Fig. 11," however, the 
"i" referred to here is different from the pixel of interest i of the reference 
frame information in the abovementioned camera motion estimation 
processing. 

When the pixel of interest i is set, the distances between this pixel of 
25 interest i and pixels that are present in the vicinity of the pixel of interest i 
(among all the pixels of all four sets of frame information) are calculated 
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(S420). Next, the closest pixel which is closest to the pixel of interest i is 
selected (S425). In this selection, for example, the coordinate values of the 
closest pixel may be stored in a specified region in the RAM. Here, if the 
coordinates of the pixel of interest i using the produced image data as a 
5 reference are designated as (xO, yO) and the coordinates of pixels present in 
the vicinity of the pixel of interest i are designates as (xf, yf), then the pixel 
which is such that [(xf - xO) 2 + (yf — yO) 2 ] 172 shows a minimum value may be 
taken as the closest pixel. 

[0066] Furthermore, frame information including the closest pixel is 

10 acquired (S430). Here, the frame information that includes the closest 
pixel may be acquired by storing this frame information in a specified region 
in the RAM, or may be acquired by storing the memory address of the frame 
information that includes the closest pixel in a specified pointer in the RAM. 

Fig. 15 shows the conditions of selection of the abovementioned 

15 closest pixel in model form. In this figure, the positions of all of the pixels 
of the reference frame information and three sets of object frame 
information are plotted, and the positions of the pixels of the image data 
that is produced (1.5 times denser in the vertical and horizontal directions) 
are also plotted. Here, a frame number f (f = 1, 2, 3, 4) is assigned to each 

20 set of frame information, and the pixel closest to the pixel of interest is 
selected from each set of frame information. In the figure, the distance 
between the selected pixel and the pixel of interest is designated as L(i, f). 
Then, fnearest(i) which is such that L(i, f) shows a minimum value is 
determined. As the pixels selected show a smaller fnearest(i) value (i. e., as 

25 the pixels selected are closer to the pixel of interest i), the jaggy in the edge 
portions of the still image tends to decrease as a result of interpolation 
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processing referring to frame information containing these pixels. In the 
case of the pixel of interest i shown in the figure, since fnearest(i) = 4, frame 
information of four frames is acquired. 

[0067] When frame information including the closest pixel is acquired, 
5 specified interpolation processing by the by-linear method or the like is 
performed using the YCbCr data (halftone data) for the closest pixel from 
this frame information, so that image data for the pixel of interest i is 
produced (S435). In this case, the image data may be produced "as is" in 
the form of YCbCr data, or image data comprising RGB may be produced by 

10 using specified calculation formulae to convert the YCbCr data into RGB 
data comprising respective RGB halftone values. 

Fig. 16 shows the conditions of interpolation processing using the 
by-linear method (co-primary interpolation method) in model form. As is 
shown in the figure, interpolation processing is performed using halftone 

15 data for four pixels 61a through 61d in the area surrounding the pixel of 
interest i (including the closest pixel 61a) from the acquired frame 
information. In the by-linear method, the weighting of the halftone data is 
gradually varied as the pixels (lattice points) 61a through 6 Id used in the 
interpolation calculations approach each other, and this variation is taken 

20 as a primary function that depends only on the halftone data of the pixels on 
both sides. Here, the regions demarcated by the four pixels 61a through 
61d that surround the pixel of interest i for which interpolation is desired 
may be divided into four demarcations by the pixel of interest i, and the 
diagonal position data may be weighted by the area ratio. 

25 If a similar procedure is performed for the respective pixels of the 

produced image data, all of the pixel values can be estimated. 
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[0068] Thus, superimposition can be performed on the basis of the 
positional relationship among the images estimated according to the camera 
motion, so that the halftone values (pixel values) of the pixels of the image 
data that is produced can be estimated from the halftone data for the pixels 
5 of the nearby frame information. Specifically, image data for the respective 
pixels is produced by interpolation, so that still images with a high image 
quality can be obtained. 

Of course, interpolation processing by the nearest neighbor method 
may be performed in order to increase the speed of the processing that 

10 produces the image data. In this case, the halftone data for the closest 
pixel i n the frame information may be used as the image data for the pixel 
of interest i. Furthermore, interpolation processing with good precision 
may be performed, as in the by-cubic method. 

Furthermore, the system may also be devised so that the "by-cubic 

15 method", "by-linear method" and "nearest neighbor method" can be 
selectively input, and so that the image data for the pixel of interest is 
produced by the image processing that is selected and input. Consequently, 
precedence can be give to either image quality or processing speed, so that 
the convenience of the system can be improved. 

20 [0069] Subsequently, a judgment is made as to whether or not image 
data has been produced by interpolation processing for all of the pixels that 
produce the image data (S440). In a case where pixels remain that have 
not produced image data, the processing of S415 through S440 is repeated 
while successively moving the pixel of interest over the pixels corresponding 

25 to the image data that is produced. In a case where image data has been 
produced for all of the pixels, this flow is ended. 
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Thus, a plurality of sets of frame information that have been 
subjected to a coordinate conversion are synthesized, so that image data 
that expresses still images in halftones by means of numerous pixels is 
produced. Furthermore, in this stage, the YCbCr data may also be 
5 converted into RGB data or the like using specified calculation formulae. 

When an actual experiment was performed, results in which the 
edge portions of the still images were free of jaggy were obtained by 
performing interpolation processing using halftone data for the pixel closest 
to the pixel of interest i (among all of the pixels of the plurality of sets of 

10 frame information). Thus, high-definition high- resolution images that 
cannot be obtained using only a single frame can be produced by using a 
plurality of frames. In particular, high-definition high-resolution images 
can be obtained in cases where the precision of the estimation of the 
movement of the individual frame images is good. 

15 [0070] As a result of the abovementioned processing, the image data 
that expresses still images is produced by synthesis form a plurality of sets 
of frame information in which translational movement deviation and 
rotational deviation among the plurality of frame images expressed by the 
plurality of sets of frame information is eliminated. Manual blurring of 

20 video imaged by means of a digital video camera or the like contains not 
only deviation comprising parallel movement in the horizontal and vertical 
directions, but also diagonal components. However, since rotational 
deviation is also reduced, manual blurring including rotational components 
can be sufficiently corrected. Accordingly, still images with a high image 

25 quality can be obtained from a plurality of sets of frame information 
contained in video information. Of course, the translational movement 



PF04J801 



components are also sufficiently corrected. Furthermore, since 
translational movement deviation of the specified central positions about 
which the object frame images are rotated is eliminated, the diagonal 
components of manual blurring can be more reliably corrected, so that still 
5 images with a higher image quality can be obtained. 

Here, since the plurality of sets of frame information that are 
synthesize constitute information that is continuous in a time series with 
little variation, still images with a high image quality can be obtained by 
means of a simple construction as a result of image data that expresses still 
10 images being produced by the synthesis of the abovementioned sets of frame 
information. 
[0071] 

(4-5) Synthesis of Frame Images - Part 2 

Furthermore, synthesis processing may also be performed in 
15 accordance with the flow chart shown in Fig. 17. 

Although this is not shown in the figures, the resolution of the still 
images is acquired and the number of pixels of the image data following 
synthesis is determined in the same manner as in S405 through S410 in Fig. 
14, and the threshold value Lth of the shortest pixel distance is then 
20 acquired (S505). For example, a printing interface screen which has a 
specified selection column may be displayed, and a parameter that 
expresses the threshold value Lth may be acquired from an operating input 
into this selection column, and stored in the RAM. 

Here, as is shown in Fig. 18, the threshold value Lth of the shortest 
25 pixel distance is the threshold value that is used to judge whether or not a 
pixel in the frame information is within a specified distance from the pixel of 
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interest i. In the example shown in the figure, Vi of the distance among the 
pixels in the image data that is produced is used as the threshold value Lth. 
The circles drawn by dotted lines that are centered on the respective pixels 
including the pixel of interest indicate regions within the distance of the 
5 threshold value Lth from the same pixels. 

Furthermore, besides acquiring a parameter that expresses the 
threshold value Lth by an operating input, it would also be possible to set 
the threshold value Lth automatically from values such as the number of 
pixels of the images produced, the total number of pixels of the frame 
10 images or the like. 

For example, if it is assumed that 

Lth = 3 x (distance among pixels in the images produced) x 
(number of pixels of the images produced) / 
(total number of pixels of the frame images) 
15 then the threshold value Lth can be set as a value which is such that an 
average of 3 frame image pixels fall within the circles shown in Fig. 18. 
[0072] Next, the position of the pixel of interest i that produces the 
image data expressing still images in halftones is set in the same manner as 
in the abovementioned S415 through S420 (S510), and the pixels of 
20 coordinates (xf, yf) that are present in the vicinity of the pixel of interest i 
and the distance [(xf — xO) 2 + (yf — y0) 2 ] 1/2 from the pixel of interest i of 
coordinates (xO, yO) are calculated (S515). 

Furthermore, the pixels of the frame information whose calculated 
distance is equal to or less than the threshold value Lth are selected (S520). 
25 In this selection, for example, the coordinate values of the pixels may be 
stored in a specified region of the RAM. In the example shown in Fig. 18, a 
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pixel with a distance of L(i, 4) and a pixel with a distance of L(i, 2) are 
selected. 

Thus, pixels which are within a specified range using the pixel of 
interest i as a reference can be selected from the plurality of sets of frame 
5 information. 

[0073] Subsequently, frame information including the selected pixels is 
acquired (S525). Next, for each set of frame information, specified 
interpolation processing such as interpolation processing by the by-linear 
method or the like is performed using YCbCr data for the closest pixel from 

10 the same frame information, so that halftone data for each set of frame 
information is produced (S530). Specifically, interpolation processing is 
performed using frame information that includes pixels within a specified 
range using the pixel of interest i as a reference (among the plurality of sets 
of frame information). 

15 Then, the arithmetical mean of the respective halftone data 

following interpolation processing is determined, so that halftone data for 
the pixel of interest i is produced (S535). Here, the halftone data following 
interpolation processing may be YCbCr data or RGB data. Besides 
arithmetical averaging, it would also be possible to use geometrical 

20 averaging, harmonic averaging, averaging with different weighting 
according to the frame information or the like in the summarization of the 
halftone data. In cases where there is only one pixel of frame information 
within the specified range from the pixel of interest i, the speed of the 
processing that produces the image data can be increased by omitting the 

25 processing that performs averaging. 

[0074] Furthermore, in cases where there is no pixel of frame 
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information with a distance equal to or less than the threshold value Lth (i. 
e., the specified range) from the pixel of image data produced, as in the case 
of the pixel iA in Fig. 18, interpolation processing may be performed using 
any frame information among the plurality of sets of frame information. In 
5 this case, interpolation processing may be performed using the reference 
frame information, or interpolation processing may be performed by 
acquiring frame information that includes the closest pixel as shown in Fig. 
14. In this way, all of the pixels of the image data that is produced can be 
securely embedded. 

10 [0075] Subsequently, a judgment is made as to whether or not image 
data has been produced by interpolation processing for all of the pixels of 
the image data that is produced (S540). In cases where pixels remain for 
which image data has not been produced, the processing of S505 through 
S540 is repeated while successively moving the pixel of interest i. If image 

15 data has been produced for all of the pixels, this flow is ended. 

As a result of the above processing, a plurality of sets of frame 
information that have been subjected to a coordinate conversion are 
synthesized, so that image data that expresses still images in halftones by 
means of numerous pixels is produced. 

20 In particular, if the abovementioned processing is performed, the 

deviation among the frame images can be made inconspicuous in cases 
where the precision of the estimation of the movement of the individual 
frame images is not good. Accordingly, in such cases, high- definition 
high-resolution images can be obtained. 

25 [0076] 

(4-6) Synthesis of Frame Images - Part 3 
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Furthermore, the synthesis processing may also be performed in 
accordance with the flow chart shown in Fig. 19. 

Although this is not shown in the figures, the resolution of the still 
images is acquired and the number of pixels of the image data following 
5 synthesis is determined in the same manner as in S405 through S410 in Fig. 
14, and the position of the pixel of interest i that produces image data that 
expresses still images in halftones is then set (S605). Next, the frame 
information (among the four sets of frame information) for which 
interpolation processing is performed is set (S610). 

10 The set frame information may be stored in a specified region of the 

RAM, or the memory address may be stored in a specified pointer. 
Furthermore, pixels surrounding the pixel of interest i are selected from the 
set frame information (S615). Then, image data for the pixel of interest i is 
produced by performing specified interpolation processing by the by-cubic 

15 method or the like using halftone data such as YCbCr data or the like for 
the selected pixels (S620). 

The bycubic method utilizes data for a total of 16 pixels, including 
not only the pixels (lattice points) surrounding the pixel of interest i for 
which interpolation is desired, but also the surrounding pixels located to the 

20 outside of these surrounding pixels. The halftone value for the pixel of 
interest i is determined while being affected by the halftone values of the 
abovementioned total of 16 pixels surrounding this pixel of interest i. For 
example, if interpolation is performed by first-order equations, then 
weighting may be added in inverse proportion to the distance from two 

25 pixels on either side of the pixel of interest i. Furthermore, the effect 
according to distance may be expressed by a third-order folding function (t) 



53 



PF04J801 



= {sin(7it)}/7it. 

[0077] Subsequently, a judgment is made as to whether or no 

interpolation processing has been performed for all of the frame information 
(S625). In cases where frame information remains for which interpolation 
5 processing has not been performed, the processing of S610 through S625 is 
repeated; if interpolation processing has been performed for all of the 
frame information, the processing proceeds to S630. 

In S630, the arithmetical mean of the respective halftone data 
following interpolation processing is determined, and image data for the 

10 pixel of interest i is produced. Besides arithmetical averaging, it would 
also be possible to use geometrical averaging, harmonic averaging, 
averaging with different weighting according to the frame information or 
the like in the summarization of the halftone data. 

Then, a judgment is made as to whether or not image data has been 

15 produced by interpolation processing for all of the pixels of the image data 
that is produced (S635). In cases where pixels remain for which image 
data has not been produced, the processing of S605 through S635 is 
repeated while successively moving the pixel of interest i. If image data 
has been produced for all of the pixels, this flow is ended. 

20 [0078] In the above processing as well, a plurality of sets of frame 
information for which a coordinate conversion has been performed are 
synthesized, so that image data that expresses still images in halftones by 
means of numerous pixels is produced. Here, since pixels are present in 
lattice form in each set of frame information, still images with a high image 

25 quality can be obtained by utilizing interpolation processing that has good 
precision such as the by-cubic method or the like. Furthermore, since the 
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deviation among frame images can be made inconspicuous in cases where 
the precision of the estimation of the movement of the individual frame 
images is not good, high-definition high-resolution images can be obtained 
in such cases. 
5 [0079] 

(5) Modifications 

The image producing device and image deviation amount detection 
device of the present invention may have various constructions. 

For example, the printer may be integrated with a computer. 

10 Besides being realized within the main body of a computer, the 
abovementioned flow may also be realized (either in part or as a whole) in a 
printer or special image output device. 

.Besides being arranged in the form of a dot matrix in which the 
pixels are lined up in order in the vertical and horizontal directions, the 

15 numerous pixels that constitute the image data may have various other 
constructions. For example, the image data may also be constructed from 
pixels that are lined up in order in the form of beehive, in which the pixels 
are densely arranged in a regular hexagonal shape. 

There is no limit on the number of the plurality of sets of frame 

20 information that are synthesized, as long as this number is two or more. 
Furthermore, the present invention can also be used in cases where image 
data that expresses still images is produced by synthesizing portions of 
frame images. Moreover, even in the case of interlaced video which has 
regions that cannot be comprehended on a line by line basis, an effect in I-P 

25 conversion can be manifested by superimposition on a field by field basis, so 
that one scene of a moving image can be expressed as a high-resolution still 
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image, and so that the definition can be increased in the case of printing. 
Of course, the present invention is also effective in the superimposition of a 
plurality of frames in which the object is not necessarily conversion of the 
resolution (e. g., panorama synthesis and the like). 
5 Furthermore, in the present invention, since translational 

movement deviation and rotational deviation can be eliminated, 
translational movement deviation and rotational deviation arising from 
causes other than manual blurring can also be corrected. 

[0080] Furthermore, in the detection of the translational movement 
10 correction amount and rotational correction amount of the frame images, 
this detection may also be performed while varying the reference frame 
information. 

Fig. 20 shows in model form the processing that is performed by an 
image producing device constituting a modification. In this modification as 

15 well, four sets of frame information are acquired from the video information. 
Then, in the camera motion estimation processing, translational movement 
amount information and rotation amount information are acquired from 
sets of frame information that are adjacent to each other in time series order. 
Specifically, first translational movement amount information (translational 

20 movement correction amounts ul and vl) and first rotation amount 
information (rotational correction amount 51) that express the deviation 
between the two images expressed by a first set of frame information 
comprising frame 1 that is used as reference frame information and second 
set of frame information comprising frame 2 that is used as object frame 

25 information are acquired. Next, second translational movement amount 
information (translational movement correction amounts u2 and v2) and 
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second rotation amount information (rotational correction amount 82) that 
express the deviation between the two images expressed by the second set of 
frame information comprising frame 2 that is used as reference frame 
information and third set of frame information comprising frame 3 that is 
5 used as object frame information are acquired. Furthermore, second 
translational movement amount information in a separate sense 
(translational movement correction amounts u3 and v3) and second rotation 
amount information in a separate sense (rotational correction amount 53) 
that express the deviation between the two images expressed by the second 
10 set of frame information in a separate sense comprising frame 3 that is used 
as reference frame information and third set of frame information in a 
separate sense comprising frame 4 that is used as object frame information 
are acquired. 

[0081] Furthermore, the camera motion estimation processing can also 

15 be performed according to the flow shown in Fig. 11. 

Specifically, reference frame information is set from a plurality of 
sets of frame information acquired from the video information (in the order 
of frames 1, 2 and 3), and object frame information is likewise set (in the 
order of frames 2, 3 and 4). Furthermore, the respective variables are 

20 initially set (S205 through S215). Next, for all of the pixels of the reference 
frame information, the positions of the pixels of interest i and i' are set from 
both sets of frame information, and Pti, ki, M20, Mil, M02, m u , m v , mg, c u , 
c v and C6 are successively calculated (S220 through S230). Furthermore, d 
is calculated, and the translational movement correction amounts ul 

25 through u3 and vl through v3 and rotational correction amounts 51 through 
53 are successively calculated and stored in memory as translational 
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movement amount information and rotation amount information (S235 
through S250). Specifically, translational movement amount information 
(ul, vl) and rotation amount information (81) expressing the deviation 
between the two images expressed by the first and second sets of frame 
5 information can be acquired on the basis of these first and second sets of 
frame information, and translational movement amount information (u2, 
u3, v2, v3) and rotation amount information (62, 83) expressing the 
deviation between the two images expressed by the second and third sets of 
frame information can be acquired on the basis of these second and third 
10 sets of frame information. When translational movement amount 
information and rotation amount information has been acquired for all of 
the frame information, this flow is ended (S255). 

[0082] When translational movement amount information and rotation 
amount information are acquired, frame conversion processing is performed 

15 for the frame information of frames 2 through 4 in accordance with the 
frame image of the fist frame 1 in time series order. 

In outline, the frame conversion processing can be performed 
according to the flow shown in Fig. 12. 

Specifically, object frame information in which the coordinates of the 

20 pixels are to be converted is set in the order of frames 2 through 4, and the 
position of the pixel of interest i' for which a coordinate conversion is to be 
performed is set (S305 through S3 10). Next, for all of the pixels of the 
object frame information, the position of the pixel of interest i' is converted 
so that the translational movement deviation and rotational deviation with 

25 respect to the frame image of frame 1 are eliminated (S315 through S325). 
Here, for the object frame information of frames 2, 3 and 4, a coordinate 
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conversion is performed so that the positions of the respective pixels of 
interest i' are caused to undergo a translational movement of — ul, -ul — u2 
and — ul — u2 — u3 in the direction of the x axis and — vl, — vl — v2 and — vl — 
v2 — v3 in the direction of the y axis, and a rotational movement of —81, -81 
5 - 82 and —81 — 82 - 83. Specifically, conversion processing can be performed 
in which the second set of frame information is converted on the basis of the 
first rotation amount information so that the deviation relative to the image 
expressed by the first set of frame information is eliminated, and in which 
the third set of frame information is converted on the basis of the first and 

10 second rotation amount information so that the rotational deviation with 
respect to the image expressed by the first set of frame information is 
eliminated. When conversion processing has been performed for the frame 
information of frames 2 through 4, this flow is ended (S330). 
[0083] Subsequently, if synthesis processing is performed, image data 

15 that expresses still images in halftones can be produced. 

Specifically, since the deviation between the images that are 
expressed by the second and third sets of frame information that are 
adjacent in time series order is generally smaller than the rotational 
deviation between the images expressed by the first and third sets of frame 

20 information, translational movement correction amounts for correcting the 
translational movement deviation and rotational correction amounts for 
correcting the rotational deviation can be detected with a higher degree of 
precision. Accordingly, translational movement amount information and 
rotation amount information can be acquired with a high degree of precision, 

25 so that still images with a higher image quality can be obtained. 

As was described above, the present invention can provide, in 
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various aspects, an image producing device, image deviation amount 
detection device, image production program and image deviation amount 
detection program which make it possible to obtain still images with a high 
image quality from a plurality of sets of frame information contained in 
5 video information. Furthermore, the present invention can also be applied 
as an image production method and image deviation amount detection 
method. 

[0084] In the abovementioned embodiments, a case was described in 
which high-resolution still images were produced by synthesizing a plurality 

10 of sets of frame information continuous in a time series that were acquired 
from video information. However, the present invention is not limited to 
this. It would also be possible to produce high-resolution images merely by 
synthesizing a plurality of sets of image information that are continuous in 
a time series. For example, such a plurality of sets of image information 

15 that are continuous in a time series may be a plurality of sets of image 
information successively acquired by a digital camera. 

Furthermore, it is not always necessary that the information used be 
a plurality of sets of image information (including frame information) that 
are continuous in a time series; this information may also be a plurality of 

20 sets of image information that are lined up in a time series. 

[0085] In the abovementioned embodiments, a personal computer (PC) is 
used as the image producing device; however, the present invention is not 
limited to this. The abovementioned image producing device may be 
contained in a video camera, digital camera, printer, DVD player, video tape 

25 player, hard disk player, camera-equipped portable telephone or the like. 
In particular, in cases where a video camera is used as the image producing 
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device of the present invention, still images with a high resolution can be 
produced from a plurality of sets of frame information contained in acquired 
video information while this video information is being acquired. Similarly, 
in cases where a digital camera is used as the image producing device of the 
present invention, still images with a high resolution can be produced from 
a plurality of sets of imaging information while imaging the object of 
imaging or checking the results of imaging. 
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